Skip to content

MILAB-6303: tree-sync introspection metrics + stop-marker follow-up fix - #1760

Merged
xnacly merged 7 commits into
mainfrom
MILAB-6303_tree-sync-metrics
Jul 27, 2026
Merged

MILAB-6303: tree-sync introspection metrics + stop-marker follow-up fix#1760
xnacly merged 7 commits into
mainfrom
MILAB-6303_tree-sync-metrics

Conversation

@xnacly

@xnacly xnacly commented Jul 21, 2026

Copy link
Copy Markdown
Member

Tree-sync instrumentation and a stop-marker follow-up fix, backing the MILAB-6303 investigation (Desktop App on slow / distant networks). Spec: milaboratory/text#190.

What this adds

Full introspection of the tree-sync load via MI_LOG_TREE_STAT, computed by reusing the state machine's existing changed flag (negligible per-resource cost):

  • Cross-cycle duplication: new / changed / unchanged resources, wasted downlink bytes, stable-metadata re-streams, BFS fetches spent on unchanged resources.
  • Per-path detail: streaming (rounds, resource frames, stop-marker frames, stop-markers that triggered a follow-up, traverse-stopped) and BFS (resources actually requested, not-found).
  • What changed each cycle: fields added / removed / value-changed, kv changes, ready flips, lock transitions, errors attached, duplicates resolved, resources marked final.

This measured that 52-76% of every polling cycle re-fetches unchanged resources (trees changed by only 2-9 resources over 3 min) on a heavy project.

Fix

The resourceTree stop-marker follow-up now propagates traverseStopRules, so the retry stops at the same boundaries instead of traversing the stop-marked subtrees unbounded. It also asserts that one follow-up round resolves every stop marker (throws if a deeper stop-marker chain remains, which would mean the retry must become a loop).

Status

Draft. Type-check clean; pl-tree unit tests pass (the one skipped failure needs a live PL_ADDRESS). No end-to-end validation of the stop-rule change under a deep tree yet; the assertion is the tripwire for that.

Greptile Summary

Adds detailed tree-synchronization introspection and changes how streaming stop markers and KV mutations are reconciled.

  • ResourceUpdateStat / TreeLoadingStat — counters describing fetched resources, duplicate traffic, mutation categories, streaming frames, BFS requests, and stop-marker follow-ups; this PR defines and populates the expanded metrics.
  • changed — the per-resource update flag controlling dataVersion, finality reevaluation, and changed-versus-unchanged statistics; this PR extends it to KV mutations but misses dynamic-field removal.
  • metadataChanged — the classifier separating structural metadata churn from value/flag-only changes; this PR introduces it for metrics but does not mark field-type transitions.
  • traverseStopRules — backend filters that stop traversal at configured resource boundaries; the initial stream applies them, while the new stop-marker follow-up loop deliberately omits them.
  • stopMarker — a streaming frame identifying a resource whose traversal was stopped and whose state may require a follow-up fetch; this PR adds multi-round resolution and related counters.
  • finalResources — locally immutable resources excluded from additional retrieval; this PR adds statistics around skipped final resources and uses the set while processing follow-up streams.
  • KV mutation — addition, replacement, or deletion of resource key/value data; this PR now marks these operations as resource changes so versions and finality are updated.

Confidence Score: 2/5

The PR should not merge until stop-marker follow-ups preserve descendant traversal boundaries and field mutations are classified consistently.

Follow-up streaming can retrieve subtrees beyond the configured stop rules, while dynamic-field removals and field-type transitions produce incorrect state-version/finality handling or misleading diagnostics.

Files Needing Attention: lib/node/pl-tree/src/sync.ts, lib/node/pl-tree/src/state.ts

Important Files Changed

Filename Overview
lib/node/pl-tree/src/sync.ts Adds detailed loading metrics and iterative stop-marker resolution, but follow-up streams bypass configured traversal boundaries.
lib/node/pl-tree/src/state.ts Adds resource mutation statistics and correctly recognizes KV changes, but field removal and field-type bookkeeping misclassify updates.
lib/node/pl-tree/src/synchronized_tree.ts Passes the loading-stat object into state application so load and mutation counters are emitted together.
.changeset/tree-sync-duplication-metrics.md Declares the pl-tree patch and summarizes the instrumentation, stop-marker loop, and KV change handling.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Construct loading request] --> B[Initial resourceTree with stop rules]
  B --> C{Frame kind}
  C -->|Resource| D[Collect resource state and metrics]
  C -->|Stop marker already final| E[Skip]
  C -->|Unknown stop marker| F[Queue follow-up seed]
  F --> G[Follow-up resourceTree without stop rules]
  G --> C
  D --> H[updateFromResourceData]
  H --> I{Resource changed?}
  I -->|Yes| J[Advance dataVersion and reevaluate finality]
  I -->|No| K[Count unchanged bytes]
Loading

Fix All in Claude Code

Prompt To Fix All With AI
Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
lib/node/pl-tree/src/sync.ts:395-398
**Follow-ups bypass traversal boundaries**

When the initial stream returns a non-locally-final stop marker for a terminal resource with referenced children, the follow-up omits `traverseStopRules` and traverses beyond that resource. This downloads and materializes descendants outside the configured boundary, potentially cascading through a large subtree and defeating the intended load reduction.

### Issue 2 of 3
lib/node/pl-tree/src/state.ts:691-700
**Field removal misses changed state**

When an existing resource loses a dynamic field without another simultaneous mutation, this branch deletes the field but leaves `changed` false. The update is consequently counted as unchanged, `dataVersion` is not advanced, and finality is not re-evaluated until a later mutation.

### Issue 3 of 3
lib/node/pl-tree/src/state.ts:614-643
**Type changes masquerade as stable metadata**

When a Dynamic field transitions to another permitted type, this branch sets `changed` but not `metadataChanged`. The aggregate classifier therefore increments `metadataStableChanged` for genuine structural churn, overstating stable-metadata re-streaming in `MI_LOG_TREE_STAT`.

Reviews (1): Last reviewed commit: "MILAB-6303: fetch stop-marker follow-up ..." | Re-trigger Greptile

Greptile also left 1 inline comment on this PR.

Context used:

xnacly added 2 commits July 20, 2026 15:47
Extend the MI_LOG_TREE_STAT counters with per-cycle re-fetch
classification computed in updateFromResourceData: new, changed and
unchanged resources, wasted downlink bytes, BFS fetches spent on
unchanged resources, and changes that re-streamed stable metadata.
Quantifies how much of each poll re-fetches resources the client
already holds. Reuses the existing changed flag, no extra work per
resource.
Expand MI_LOG_TREE_STAT into full per-path introspection: streaming
detail (rounds, resource/stop-marker frames, follow-up seeds,
traverse-stopped), BFS detail (resources requested, not found), and a
per-cycle change breakdown (fields added/removed/changed, kv, ready
flips, locks, errors, duplicates resolved, resources marked final),
alongside the existing cross-cycle duplication counters.

Fix the resourceTree stop-marker follow-up to propagate traverseStopRules
so the retry stops at the same boundaries instead of traversing the
stop-marked subtrees unbounded. Assert that one follow-up round resolves
every stop marker (throws if a deeper stop-marker chain remains, which
would mean the retry must become a loop).
@changeset-bot

changeset-bot Bot commented Jul 21, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8b226cf

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages
Name Type
@milaboratories/pl-tree Patch
@milaboratories/pl-drivers Patch
@milaboratories/pl-middle-layer Patch
@platforma-sdk/test Patch
@platforma-sdk/pl-cli Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@notion-workspace

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request instruments tree-sync statistics (MI_LOG_TREE_STAT) to provide detailed introspection on cross-cycle duplication, per-path details, and cycle changes, while also fixing stop-marker follow-up propagation. The review comments correctly identify a critical issue where adding, modifying, or deleting KV entries does not set the changed flag to true. This omission causes incorrect state versioning, inaccurate introspection metrics, and prevents proper final state evaluation.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread lib/node/pl-tree/src/state.ts
Comment thread lib/node/pl-tree/src/state.ts
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 53.64%. Comparing base (5a98dbe) to head (8b226cf).
⚠️ Report is 43 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1760      +/-   ##
==========================================
+ Coverage   53.49%   53.64%   +0.15%     
==========================================
  Files         365      366       +1     
  Lines       19524    19679     +155     
  Branches     4291     4341      +50     
==========================================
+ Hits        10444    10557     +113     
- Misses       7792     7818      +26     
- Partials     1288     1304      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

xnacly added 4 commits July 22, 2026 10:16
KV mutations previously did not set the changed flag, so they never
bumped dataVersion, skipped isFinalPredicate re-evaluation, and were
miscounted as unchanged re-fetches in the tree-sync stats. Set changed
on KV add, modify, and delete so versioning, finality, and the
duplication metrics are correct. Addresses PR review.
Comment out the throw on unresolved stop-marker seeds after the follow-up
round while the deeper-chain case is under investigation. Keep the
traverseStopRules propagation and leave re-enable instructions inline.
…hains

Propagating traverseStopRules to the follow-up bounded the retry but left
stop-marker chains deeper than one level unresolved: a loaded resource
referenced a deeper stop-marked resource that was never fetched, so
updateFromResourceData threw "orphan resource" on open. Loop the
follow-up (each round with the same stop rules, deduping fetched ids)
until no stop-marker seeds remain, so everything referenced is loaded.
Replaces the disabled one-shot assertion.
Reapplying traverseStopRules on the follow-up re-flagged the very
resources being retried as stop markers (the rule is finality-based and
they are server-final), so their state never loaded and resources
referencing them threw "orphan resource" on open. Fetch follow-up seeds
plainly; keep the loop only to resolve any residual stop markers. Update
the changeset accordingly.
@xnacly
xnacly marked this pull request as ready for review July 27, 2026 12:10
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Comment thread lib/node/pl-tree/src/sync.ts
Comment thread lib/node/pl-tree/src/state.ts Outdated
Comment thread lib/node/pl-tree/src/sync.ts Outdated
Per review: replace the positional boolean arg on updateFromResourceData with
an options object ({ allowOrphanInputs, stat }), updating the one caller; and
build formatTreeLoadingStat as a single template literal instead of concatenating.
@xnacly
xnacly enabled auto-merge July 27, 2026 13:35
@xnacly
xnacly added this pull request to the merge queue Jul 27, 2026
Merged via the queue into main with commit 0da9036 Jul 27, 2026
14 checks passed
@xnacly
xnacly deleted the MILAB-6303_tree-sync-metrics branch July 27, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants